A discriminative reliability-aware classification model with applications to intelligibility classification in pathological speech

Many computational paralinguistic tasks need to work with noisy human annotations that are inherently challenging for the human annotator to provide. In this paper, we propose a discriminative model to account for the inherent heterogeneity in the reliability of annotations associated with a sample while training automatic classification models. Reliability is modeled as a latent factor that governs the dependence between the observed features and its corresponding annotated class label. We propose an expectation-maximization algorithm to learn the latent reliability scores using maximum entropy models in a mixture-of-experts like framework. In addition, two models a feature dependent reliable model and a feature independent unreliable model are also learned. We test the proposed method on classifying the intelligibility of pathological speech. The results show that the method is able to exploit latent reliability information on feature sets that are noisy. Comparing against a baseline of reliability-blind maximum entropy model, we show that there is merit to reliability-aware classification when the feature set is unreliable.